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Amendment to the Claims : 

This listing of claims replaces all prior versions, and 
listings, of claims in the application: 

1. (original) A method comprising: 

generating a three dimensional model of an environment from 
range sensor information representing a height field for the 
environment ; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery. 

2. (currently amended) A ^Fhe method of claim 1, 
comprising : 

generating a three dimensional model of an environment from 
range sensor information representing a height field for the 
environment ; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 
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projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein generating the three dimensional model comprises 
identifying a structure in the range sensor information, 
identifying different sections of the structure, selecting 
geometric primitives for the different sections of the structure 
based at least in part on input from a person regarding 
different shapes of the different sections, and parametric 
fitting of the geometric primitives to the range sensor 
information. 

3. (currently amended) The method of claim 2, wherein 
generating the three dimensional model further comprises: 

identifying a structure in the range sensor information; 

the selecting a set of geometric primitives comprises 
selecting geometric primitives from a group [[,]] including a 
sphere primitive and a cuboid primitive; and supcrquadratic , — fee 
use in the parametric fitting with reopect to the identified 
structure based on a shape of the otructurc 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
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information . 

4. (original) The method of claim 2, wherein the at least 
one image sensor comprises multiple image sensors, and 
generating the three dimensional model further comprises 
refining the three dimensional model based on object surfaces 
mapped from images acquired by the image sensors. 

5. (original) The method of claim 1, wherein generating 
the three dimensional model further comprises projecting and 
resampling points in the range sensor information onto a regular 
grid at a user-defined resolution to produce the height field. 

6. (original) The method of claim 5, wherein generating 
the three dimensional model further comprises processing the 
height field using hole-filling and tessellation to generate a 
triangle mesh representation of the three dimensional model. 

7. (original) The method of claim 2, wherein the range 
sensor information comprises light detection and ranging (LIDAR) 
sensor data from an active airborne laser sensor. 

8. (original) The method of claim 1, wherein tracking 
orientation information comprises tracking position and 
orientation information of the at least one image sensor by 
estimating a camera pose based at least in part on three 
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dimensional parameters of point and line structures visible in 
the camera image, and projecting real-time video imagery 
information comprises projecting real-time video imagery 
information from the at least one image sensor onto the three 
dimensional model based on the tracked orientation information. 

9. (original) The method of claim 8, wherein tracking 
position and orientation information of the at least one image 
sensor further comprises processing data from a tracking sensor 
system that integrates visual input, global navigational 
satellite system receiver input, and inertial orientation sensor 
input . 

10. (original) The method of claim 1, wherein the at least 
one image sensor comprises multiple image sensors, and 
projecting the real-time video imagery information comprises 
projecting multiple . video streams from the multiple image 
sensors onto the three dimensional model . 

11. (original) The method of claim 1, wherein projecting 
the real-time video imagery information comprises: 

generating a depth map image from a video sensor viewpoint; 

and 

projective texture mapping the real-time video imagery 
information onto the three dimensional model conditioned upon 
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visibility as determined from the generated depth map image. 

12. (currently amended) A The method of claim 11/ 
comprising : 

generating a three dimensional model of an environment from 
range sensor information representing a height field for the 
environment ; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein projecting the real-time video imagery information 
comprises generating a depth map image from a video sensor 
viewpoint, and projective texture mapping the real-time video 
imagery information onto the three dimensional model conditioned 
upon visibility as determined from the generated depth map 
image ; and 

wherein generating the depth map image and projective 
texture mapping the real-time video imagery information are 
performed using a one -pass approach on graphics hardware that 
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13. (original) The method of claim 1, wherein visualizing 
the three dimensional model comprises: 

video-projecting onto a display screen using a stereo 
video-projector; and 

coupling a rendering viewpoint to a user's head position 
using data from a tracker. 

14. (original) An augmented virtual environment system 
comprising: 

a model construction component that generates a three 
dimensional model of an environment from range sensor 
information representing a height field for the environment; 

a dynamic fusion imagery projection component that projects 
real-time video imagery information from at least one image 
sensor onto the three dimensional model based on orientation 
information of the at least one image sensor tracked in the 
environment with respect to the three dimensional model in real- 
t ime ; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery. 

15. (original) The system of claim 14, wherein the at 
least one image sensor comprises multiple image sensors. 
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16. (original) The system of claim 15, wherein the real- 
time video imagery information comprises pre-recorded real-time 
video imagery information. 

17. (original) The system of claim 15, wherein the dynamic 
fusion imagery projection component bases the real-time video 
imagery projection on a viewpoint separate from viewpoints 
associated with the multiple image sensors. 

18. (currently amended) An The augmented virtual 
environment system of claim 14, comprising: 

a model construction component that generates a three 
dimensional model of an environment from range sensor 
information representing a height field for the environment; 

a dynamic fusion imagery projection component that projects 
real-time video imagery information from at least one image 
sensor onto the three dimensional model based on orientation 
information of the at least one image sensor tracked in the 
environment with respect to the three dimensional model in real- 
time; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the model construction component performs 
operations comprising : 
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identifying a structure in the range sensor information; 

identifying different sections of the structure; 

selecting a oct of geometric primitives , including a 
oupcrquadratic , for the different sections of the structure 
based at least in part on input from a person regarding 
different shapes of the different sections a ohapc of the 
otructurc ; and 

parametric fitting of the selected geometric primitives to 
the range sensor information. 

19. (original) The system of claim 18, wherein the 
operations further comprise: 

projecting and resampling points in the range sensor 
information onto a regular grid at a user-defined resolution to 
produce the height field; and 

processing the height field using hole-filling and 
tessellation to generate a triangle mesh representation of the 
three dimensional model. 

20. (original) The system of claim 14, wherein the at 
least one image sensor comprises multiple image sensors, and the 
system further comprises a model refinement component that 
refines the three dimensional model based on object surfaces 
mapped from images acquired by the image sensors . 
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21. (original) The system of claim 14, further comprising 
a tracking sensor system that integrates visual input, global 
navigational satellite system receiver input, and inertial 
orientation sensor input to obtain position and orientation 
information of the at least one image sensor, and the dynamic 
fusion imagery projection component projects the real-time video 
imagery information based on the position and orientation 
information. 

22. (currently amended) An The augmented virtual 
environment system of claim 14, comprising : 

a model construction component that generates a three 
dimensional model of an environment from range sensor 
information representing a height field for the environment; 

a dynamic fusion imagery projection component that projects 
real-time video imagery information from at least one image 
sensor onto the three dimensional model based on orientation 
information of the at least one image sensor tracked in the 
environment with respect to the three dimensional model in real- 
time; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the visualization sub-system comprises the dynamic 
fusion imagery projection component and graphics hardware that 
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supports SGI OpenGL extensions, and uses a one-pass approach on 
the graphics hardware to generate a depth map image from a video 
sensor viewpoint and projective texture map the real-time video 
imagery information to project the real-time video imagery 
information conditioned upon visibility as determined from the 
generated depth map image. 

23. (original) The system of claim 14, wherein the 
visualization sub-system comprises: 

a stereo video-projector; and 
a tracker. 

24. (cancelled) 

25. (currently amended) The method of claim 2j3 34-, wherein 
the surface comprises a two dimensional surface. 

26. (original) The method of claim 25, wherein placing the 
two dimensional surface comprises: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
^ and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 



12 



Serial No.: 10/676,377 Attorney's Docket No .: 06666/156001 

three dimensional model, and the moving region. 

27. (cancelled) 

28. (cancelled) 

29. (currently amended) A The method of claim 2 8 , 
comprising ; 

obtaining a three dimensional model of an environment; 

identifying in real time a region in motion with respect to 
a background image in real - time video imagery information from 
at least one image sensor having associated position and 
orientation information with respect to the three dimensional 
model, the background image comprising a single distribution 
background dynamically modeled from a time average of the real- 
time video imagery information; 

placing a surface that corresponds to the moving region in 
the three dimensional model; 

projecting the real-time video imagery information onto the 
three dimensional model, including the surface, based on the 
position and orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein identifying a region in motion in real time 
comprises subtracting the background image from the real-time 
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video imagery information, identifying a foreground object in 
the subtracted real-time video imagery information, validating 
the foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 

wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
a noise filter; 

wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
background image as a temporal pixel average of at least five 
recent image frames in the real-time video imagery information. 

30. (currently amended) The method of claim 2_9 34-, further 
comprising tracking the position and orientation information of 
the at least one image sensor in the environment with respect to 
the three dimensional model in real-time. 

31. (original) The. method of claim 30, wherein obtaining a 
three dimensional model of an environment comprises generating 
the three dimensional model of the environment from range sensor 
information representing a height field for the environment. 

32. (cancelled) 
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33. (currently amended) The system of claim 37 wherein 
the surface comprises a two dimensional surface. 

34. (original) The system of claim 33, wherein the object 
detection and tracking component places the two dimensional 
surface by performing operations comprising: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

35 . (cancelled) 

36 . (cancelled) 

37. (currently amended) An ¥he augmented virtual 
environment system of claim 3 6 , comprising: 

an object detection and tracking component that identifies 
in real time a region in motion with respect to a background 
image in real-time video imagery information from at least one 
image sensor having associated position and orientation 
information with respect to a three dimensional model, the 
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background image comprising a single distribution background 
dynamically modeled from a time average of the real-time video 
imagery information, and places a surface that corresponds to 
the moving region with respect to the three dimensional model; 

a dynamic fusion imagery projection component that projects 
the real-time video imagery information onto the three 
dimensional model, including the surface, based on the position 
and orientation information; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the object detection and tracking component 
identifies the moving region by performing operations comprising 
subtracting the background image from the real-time video 
imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the 
foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 

wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
a noise filter; and 

wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
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background image as a temporal pixel average of at least five 
recent image frames in the real-time video imagery information. 

38. (currently amended) The system of claim 37 *2-, further 
comprising a tracking sensor system that integrates visual 
input, global navigational satellite system receiver input, and 
inertial orientation sensor input to obtain the position and the 
orientation information associated with the at least one image 
sensor in real time in conjunction . with the real-time video 
imagery. 

39. (original) The system of claim 38, further comprising 
a model construction component that generates the three 
dimensional model of an environment from range sensor 
information representing a height field for the environment. 

40. (original) A machine -readable medium embodying 
information indicative of instructions for causing one or more 
machines to perform operations comprising: 

generating a three dimensional model of an environment from 
range sensor information representing a height field for the 
environment ; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 
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projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery. 

41. (currently amended) A 3*ie machine -readable medium e# 
claim 40 ; embodying information indicative of instructions for 
causing one or more machines to perform operations comprising: 

generating a three dimensional model of an environment from 
range sensor information representing a height field for the 
environment ; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model- with the projected 
real-time video imagery; 

wherein generating the three dimensional model comprises: 

identifying a structure in the range sensor information; 

identifying different sections of the structure; 

selecting a oct of geometric primitives-? — including a 
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oupcrquadratic , for the different sections of the structure 
based at least in part on input from a person regarding 
different shapes of the different sections a ohapc of the 
structure ; and 

parametric fitting of the selected geometric primitives to 
the range sensor information. 

42. (cancelled) 

43. (cancelled) 

44. (cancelled) 

45. (new) The method of claim 3, wherein 

the selecting geometric primitives comprises selecting 
geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 

46. (new) The system of claim 18, wherein 

the selecting geometric primitives comprises selecting 
geometric primitives from a group including a sphere primitive 
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and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information. 

47. (new) The system of claim 46, wherein 

the selecting geometric primitives comprises selecting 
geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 

48. (new) The machine -readable medium of claim 41, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from a group including a sphere primitive 
and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information. 

49. (new) The machine-readable medium of claim 48, wherein 
the selecting geometric primitives comprises selecting 



20 



Serial No.: 10/676,377 Attorney's Docket No .: 06666/156001 

geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 
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